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ABSTRACT 



The Hogg Foundation for Mental Health created the School of 
the Future (SoF) project to enable selected Texas schools to coordinate and 
implement school -based social and health services on their campuses and to 
demonstrate the effectiveness of this method of service delivery by 
evaluating the project to show the process used and the outcomes that 
resulted from these efforts. This booklet was developed to accompany another 
document, "Beyond the Classroom, " which focuses exclusively on the 
qualitative, or process, evaluation of the SoF project. This booklet 
describes the decision-making processes involved and many of the lessons 
learned by project staff while evaluating the project. The first section, 

"The School of the Future Evaluation Process, " is a rather detailed 
description of the SoF project and its evaluation strategies. The second 
section, "Challenges Encountered and Lessons Learned, " describes the lessons 
learned from the SoF evaluation. Whenever possible an example has been 
included to give the lessons some context. This document does not contain any 
discussion of the results of the evaluations. It is intended to aid those who 
are developing school -based services projects who want guidance in developing 
a methodologically sound evaluation. Lessons learned stress the importance of 
defining and agreeing on evaluation goals, learning from unanticipated 
evaluation findings without diluting the evaluation by collecting information 
with no bearing on the program evaluation, and disseminating evaluation 
findings so that they can be used for program improvement. Four appendixes 
present the SoF concept paper and consent forms for parents, teachers, and 
students participating in the evaluation. (Contains 2 tables and 27 
references.) (SLD) 
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PREFACE 



W hen the Hogg Foundation fot Mental Health cteated the School of 
the Futute (SoF) ptoject, it had two majot goals: (1) to enable 
selected Texas schools to cootdinate and implement school-based 
social and health setvices on theit campuses and (2) to demonsttate 
the effectiveness of this method of setvice delivety by evaluating the pto- 
ject to show both the ptocess used and the outcomes that tesulted ftom 
these effotts. 

The ptoject was funded by the Foundation fot five yeats, 1990 to 1995. 
Based on the fact that it has been continued in the sixth and seventh yeats 
under other funding at each of the four sites — Austin, Dallas, Houston, 
and San Antonio — and that it has been incorporated into the school sys- 
tems in two of these cities, it would appear that the SoF has been a remark- 
able success. 

But too often, social service programs do not provide the documentation 
to substantiate the need for long-term continuation or replication. Aware 
of this, when the project began the Foundation provided an equal amount 
of funding for five years to conduct a two-fold evaluation: quantitative, 
to verify and confirm the impact of the project on the students, their fam- 
ilies, and the schools; and qualitative, to show the process of project 
development. 

Throughout the funding period, researchers used a combination of stan- 
dardized instruments, surveys, and interviews to gather data on the schools, 
the students and their families, and the services provided. The results of the 
data collected from the administration of these instruments are not report- 
ed here. That information will be included in a book to be published in the 
near future. 

This booklet was developed to accompany an earlier document. Beyond the 
Classroom, which focused exclusively on the qualitative, or process, evalu- 
ation of the SoF project. Challenges and Realities, however, describes the 
decision-making processes involved and many of the lessons learned by the 



project staff while evaluating the project. If, by imparting what we have 
learned from our experiences, we can help others improve the lives of chil- 
dren and their families, we will consider our efforts worthwhile. 



— Wayne H. Holtzman, Ph.D. 
Hogg Foundation for Mental Health 



W hen the Commission on the Mental Health of Adolescents and Young 
Adults challenged the Hogg Foundation to fund a school-based services 
demonstration project, Foundation staff realized the importance of 
incorporating an evaluation component in the project. Thus, the deci- 
sion was made to set aside an amount of money for the evaluation that was 
equal to the amount committed to the project itself. This was a relatively easy 
decision to make. The harder decision was coming up with an evaluation 
design appropriate for a multiyear, multisite demonstration project. 

Basing health and social services in the schools was not a new concept in 
1990, when the challenge was presented. Two school-based service projects 
in particular, the School Development Program and the School of the 21st 
Century, created by Drs. James Comer and Edward Zigler, respectively, had 
been implemented several years before the Foundation’s initiative. 
Evaluations of school-based service projects, however, were few and far 
between. There were no evaluation models to look at or blueprints to follow. 
This was the environment in which we set out to develop an evaluation strat- 
egy for the School of the Euture project. 

Many key decisions regarding the evaluation were not made solely by staff at 
the Hogg Eoundation. Included in the evaluation planning were faculty mem- 
bers from the University of Texas at Austin, the University of Texas Health 
Science Center at Houston, the University of Texas Health Science Center at 
San Antonio, and Southern Methodist University in Dallas. Also participat- 
ing in planning and strategy meetings were representatives from the four 
school districts in which project demonstration sites were located and direc- 
tors of several non-profit agencies. This enabled staff at the Eoundation to get 
input from a variety of sources before making final decisions about the kinds 
of data to collect, analyze, and report. 

The purpose of this document is not to report the results of the data collected 
and analyzed over the five years of the project. Instead, this booklet has been 
created to share the experiences of the evaluation of the School of the Euture 
project — what went well and what went not-so-well; which decisions turned 



out to be good and which decisions needed to be revised. Our hope is that by 
telling the story of our successes and our failures, those interested in evaluat- 
ing similar kinds of projects will be able to make fewer mistakes. One of our 
goals is to help people so they do not have to reinvent the wheel when design- 
ing their own evaluations. 

The first section of this booklet is a rather detailed description of the School of 
the Future project. We thought it important that the reader fully understand the 
project in order to comprehend the factors that went into our evaluation design. 
For this reason, this section discusses the purpose of the project, the sites 
involved, and how and what decisions were made regarding evaluation strategies. 

The second section describes the lessons learned from the School of the 
Future evaluation. These lessons are highlighted in red and can be viewed as 
recommendations or suggestions to those designing their own evaluations. 
Whenever possible, an attempt has been made to include examples from the 
project to give the lessons some context. 

Conspicuously absent from this document is any discussion on the results of 
the various surveys administered and data collected. As mentioned earlier, 
this omission is intentional. First, these results are already available in other 
SoF documents or will be presented in a book to be published in the near 
future. Second, the present document is designed for grant writers, service 
providers, and others who are already convinced of the importance and effi- 
cacy of school-based services but want guidance in developing a methodolog- 
ically sound evaluation to include in their projects. 

Our primary objective is to inform the reader on how and why various deci- 
sions were made regarding the measurement of various types of data. Another 
objective is to offer some assistance to those interested in developing their 
own evaluation designs. As the process of designing evaluations for school- 
based services continues to be documented and shared, future studies should 
benefit from the improved methodology used and, in turn, yield more effec- 
tive and useful evaluations. 
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— Scott S. Keir, Ph.D. 
— Susan Millea, Ph.D. 
Hogg Foundation for Mental Health 



PHRT I 



THE SCHOOL OF THE FOTOHE EVHLOHTION PROCESS 



I n December of 1987 in Austin, Texas, the first meeting was held of the 
Commission on the Mental Health of Adolescents and Young Adults, spon- 
sored by the Hogg Foundation for Mental Health. The 18 commission mem- 
bers met quarterly for three years. They stated as their mission: 1) to study 
issues related to the mental health of adolescents and young adults and 2) to rec- 
ommend action, in both the public and private sectors, to resolve these issues. 

In 1990, in its final report, the commission recommended that: 

Independent school districts and local agencies and organizations that 
provide youth services should collaborate in the development and 
implementation of school-based and school-linked counseling and 
other mental and physical health services for students of every grade 
level and their families. (Commission on the Mental Health of 
Adolescents and Young Adults, 1990: p. 14) 

The final recommendations of the commission were based on research by 
members and staff on school-based service projects that were underway in 
other parts of the country (Comer, 1980, 1988; Wang, 1990; Zigler, 1989). Of 
special interest were three projects that the study director for the commissions 
visited: two in New Haven, Connecticut, and one in Philadelphia, 
Pennsylvania. These projects had been in existence for several years and were 
already showing some promising results for participating children and their 
families. Directors of these programs — Dr. Edward Zigler (School of the 21st 
Century), Dr. James Comer (School Development Program), and Dr. 
Margaret Wang (Center for Education in the Inner Cities) — invited the 
study director to visit their projects and discuss issues with project staff that 
were related to the implementation of a similar initiative in Texas. 



The report of the study director to the commission members, along with other 
research by the commission on school-based service delivery, resulted in the 
commission members recommending this type of project in their report. The 
commission concluded: 



Educators and service providers are recognizing the viability of address- 
ing these interrelated problems in the context of a learning environ- 
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ment. . . . School-based and school-linked health and mental health 
services are based on the premise that if young people won’t come to 
the existing services, the services will come to them. Most adolescents 
attend school most of the time. There is greater likelihood that they 
will obtain the assistance they need ... if they have easy access to ser- 
vices. These services can be effective in preventing problems as well as 
in intervening after a youngster is in trouble, and they can help trou- 
bled youth remain in school and improve their academic standing by 
helping them cope with their problems. (Commission on the Mental 
Health of Adolescents and Young Adults, 1990: p. 14) 

The commission challenged the Hogg Foundation to develop several pilot 
projects in the state of Texas. Further, the commission believed it was impor- 
tant for these pilot projects to include a high-quality evaluation component. 
The final report stated: 

Evaluations of school-based service delivery efforts, based on sound 
data, are essential. They are also scarce. New efforts should be pilot- 
tested at designated sites, with outside evaluation incorporated into 
the program to determine the effectiveness of the process; the reaction 
of different populations involved including school personnel, service 
providers, and families and students themselves; and overall program 
success. Many innovative programs have been deemed successful by 
both administrators and clients, but without an evaluation the effec- 
tiveness of these efforts cannot be validated. (Commission on the 
Mental Health of Adolescents and Young Adults, 1990: p. 15) 

The executive committee of the Hogg Foundation agreed with the 
Commission on the Mental Health of Adolescents and Young Adults on the 
importance of school-based service delivery and, believing this type of project 
to be a funding priority for the Foundation, accepted the challenge of the com- 
mission. The Foundation committed a total of one million dollars to develop- 
ing demonstration programs in four Texas cities — Austin, Dallas, Houston, and 
San Antonio — and called the project “The School of the Future.”* Each 
demonstration site received its initial grant of $50,000 per year for five years in 
the fall of 1990. The Foundation set aside an equal amount to provide each 
site with technical assistance and evaluation support (Holtzman, 1992). 



'The project was originally called the “School of Tomorrow” but had to be renamed when it 
was discovered that this name was already being used and copyrighted by a private computer 
training school in the Dallas-Fort Worth area. In order to avoid any legal problems, the 
Foundation changed the name to the “School of the Future" project. 



DEVELOPING THE PROJECT 



The Demonstration Sites 

In order to increase the likelihood of success at the sites participating in the 
project, each of the school superintendents in the four cities was asked to help 
the Foundation identify school sites that might be interested in developing a 
school-based service project. The Hogg Foundation (HF) asked each super- 
intendent to suggest a middle school and one or two “feeder” elementary 
schools so representatives from the Foundation could meet with the princi- 
pals, teachers, and parents from those campuses. 

The Foundation hosted luncheon meetings with school and parent represen- 
tatives in each city. At each of the meetings, a vision of the project was pre- 
sented through a concept paper developed by HF staff. School representatives 
were then asked if they would be interested in being involved in such a pro- 
ject if HF were to fund it for a five-year period. Only one of the principals of 
the ten schools approached declined to participate.^ 

All four of the sites selected were located in disadvantaged communities with 
limited resources. There were, however, some differences across the sites. 
Following are brief descriptions that offer snapshots of the sites selected as the 
project officially began in the fall of 1990.^ (See Tables 1 and 2 for a demo- 
graphic breakdown of elementary and middle school students at each SoF site.) 

Austin. The Austin school site was located in the far southeast part of the 
city in an area known as Dove Springs, a neighborhood that was created in the 
1970s and had a high population turnover. Because of the newness of the 
development and the high percentage of rental units, few community services 
had been established. 

The two schools participating in the project, Mendez Middle School and 
Widen Elementary School, are located on opposite sides of the area’s main 
street and were both built around 1987. The enrollment at Mendez was 1,144 
students in 1990; Widen’s, 972. 



^The principal of another elementary school which also fed into the participating middle 
school was approached and informed about the SoF project. When she was invited to par- 
ticipate, she accepted the invitation. 

^The student enrollment and other data reported here represent figures from the beginning of 
the project (1990-91 school year). These figures have changed over the five-year period of 
the project, but not dramatically. 
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While school authorities indicated that gangs were not any more prevalent at 
Mendez Middle School than at other schools in the district, it was common 
for parents to relocate elsewhere to avoid sending their children to Mendez. 
The schools were ethnically diverse, with about half of the students Hispanic 
(primarily Mexican-origin), one-quarter African-American, and one -quarter 
Anglo (non-Hispanic white). Over half of the students were from low- 
income families, as determined by participation in the reduced/free lunch pro- 
gram in the school. 

Dallas. McMillan and Patton Elementary Schools and the Head Start pro- 
gram are located in Nolan Estes Plaza, a multipurpose facility owned by the 
Dallas Independent School District (DISD) that is located in the Oak Cliff 
area south of the Trinity River and downtown Dallas. The Plaza was a shop- 
ping center until the early 1980s, when it was converted into a school facility 
and DISD administrative offices. Also serving this site was Boude Storey 
Middle School, a large facility that was built in 1932 as the first junior high 
school in Dallas. 



Approximately three-quarters of the students served by the schools in the 
Dallas demonstration site in 1990 were African-American, and an equal per- 
centage of children received reduced/free lunches. However, there was some 
range of income levels in the community. Enrollment at McMillan 
Elementary School, which serves prekindergarten through third grade, was 
324. Most of these students move on to Patton Elementary, which had an 
enrollment of 207 fourth through sixth graders, and then to Boude Storey 
Middle School, with an enrollment of 897 pupils in grades seven and eight. 

Low-income families in the neighborhood faced a number of difficulties, 
among them unemployment, substance abuse, inadequate child care resources, 
deficient medical care, lack of access to recreation, and teen pregnancy. Not 
only were there very few local services to combat these problems, but also 
many families lacked the resources necessary to seek help. Families tended to 
be distrustful of outsiders, in part because of unkept promises in the past by 
social service agencies. Parents were minimally involved in the schools.'^ 

Houston. Three schools in an area called “the Heights” formed the nucleus 
of Houston’s School of the Future. The Heights lies northwest of downtown 



“^The Dallas SoF site declined participation in the quantitative evaluation because school dis- 
trict representatives believed administration of the survey instruments selected would nega- 
tively impact parental support of the project. As a result, this site is not included in the dis- 
cussion regarding quantitative data. 
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along both sides of Interstate Highway 10. The neighborhood was, and has 
remained, predominantly Hispanic and very poor. The demonstration site 
included Hogg Middle, Brock Elementary, and Memorial Elementary Schools. 

There were 1,065 students at Hogg Middle School, 89 percent of whom were 
Hispanic. Over three-quarters of the children qualified for the federal 
reduced/free lunch program and more than 30 percent came from single-par- 
ent homes. Half the children were overage for their grade. A survey con- 
ducted at the beginning of the project indicated that 80 percent of the chil- 
dren had one or more family members using drugs. 

Hogg Middle School was already the site of a highly successful dropout pre- 
vention program and was receptive to other outside efforts to help children. 
The dropout program served students classified as “at risk,” a category which 
included 70 percent of the school’s children. In addition to counseling, the 
dropout prevention effort included drug abuse treatment and prevention ser- 
vices. 

Memorial Elementary was built in 1926 and had a population of 419 students.^ 
Some 93 percent were Hispanic, and over 90 percent participated in the 
reduced/free lunch program. Brock Elementary, built in 1966 on land donat- 
ed by a successful African-American businessman from the Heights, enrolled 
just 307 students, of whom 30 percent were African-American and 99 percent 
were in the federal reduced/free lunch program. Both schools feed into Hogg 
Middle School. 

San Antonio. The neighborhood of the San Antonio demonstration schools 
was almost entirely Hispanic and faced numerous obstacles to mental health 
and stability. Physical boundaries isolated the neighborhood from the rest of 
the city. Throughout the years of the project, few industries or substantial 
employment opportunities existed in the area. Almost all of the school chil- 
dren lived in poverty; crime, family disorganization, and substance abuse were 
common, malnutrition and maltreatment persistent, and local service agen- 
cies overloaded with problems to remedy. 

The J.T Brackenridge Elementary School had no viable PTA or corporate 
sponsors, and the school district was financially strained. As a result, school 
personnel were frequently as involved in helping children and families solve 
social problems as in educating them. 



^Renovations completed in 1992 brought another 100 children to Memorial. 

O 



Located in the San Antonio Independent School District (SAISD), the state’s 
fourth'largest school district, J.T. Brackenridge Elementary School is a tela- 
tively modern facility serving 832 students in 1990, preschool through grade 
five. It is located in the middle of the Alazan- Apache Courts housing project, 
which was dedicated by Eleanor Roosevelt in 1939 and is one of the oldest 
housing projects in the country. J.T. Brackenridge was, and continues to be, 
almost completely Hispanic and very poor.^ 

The Alazan- Apache Courts housed over 1,000 families and 4,300 residents in 
1990. More than half of the families with children were one-parent families. 
Families in the Courts, which represented 18.6 percent of the total public- 
housing population in San Antonio and 15 percent of the children living in 
the public-housing units, had an average income of just under $5,000. 

Also serving this area was Tafolla Middle School, another large, modern facil- 
ity. With an enrollment of 961 students in grades 6 through 8, Tafolla offered 
both mainstream educational courses and a magnet language arts/accelerated 
studies program. Almost all of J.T. Brackenridge graduates attended Tafolla, 
though a relatively small number were enrolled in the accelerated language- 
arts program. Tafolla also was 90 percent Hispanic, with almost 84 percent of 
the students participating in the reduced/free lunch program. 

The Project Concept 

The concept paper which was distributed to the sites by the Hogg Foundation 
was a skeleton of what the school-based service project should look like after a 
few years into the grant period. (See Appendix A for concept paper distributed 
to site representatives.) The representatives at the demonstration sites selected 
for these HE grants were asked to tailor this concept to their respective sites, 
based on the needs and available resources in their schools and communities. 

Basically, the concept was to provide an integrated array of both treatment 
and preventive services, using the neighborhood schools as the locus for their 
delivery. Key aspects of the project included the following: 

• Integration of multiple services . Few social problems exist in isolation. 
Children suffering from child abuse are more likely to experience other 
forms of domestic violence in their homes, family involvement in sub- 




^Another school located in the Alazan-Apache Courts area, De Zavala Elementary, was added 
as a pilot site a year after the project began in San Antonio. Like J.T. Brackenridge 
Elementary, this school was almost totally Hispanic and very poor. 
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stance abuse, inadequate parental discipline and supervision, and so forth. 
Single-service approaches must refer clients to other agencies, usually at 
other locations, when problems turn out to be multifaceted. A coordi- 
nated approach, which includes referral and follow-up, is more likely to 
succeed when integrated, multiple services are offered at a single site. 

• The school. Using the school as the locus of services offers many advan- 
tages. These include: 

Facilities. School space is underutilized. In the early morning, late after- 
noon, evening, and on the weekend, a school’s physical plant could be 
available for meetings, presentations, treatment, and other types of services. 
Identification and outreach. In the school, children are observed each day, 
and their appearance and behavior often provide indications of problems 
at home. For the same reason, the follow-up of children after service 
delivery is much easier in the school environment. 

Image. Schools generally have a good image in their neighborhood, and 
families trust school personnel. Frequently, parents and children turn to 
the school principal, counselors, and teachers for advice about medical, 
psychological, marital, financial, and other “non-educational” problems. 
Location. For low-income families, transportation is frequently a major 
difficulty in accessing programs designed to help them. For families with- 
out a car, travel to a centrally located agency may be costly and time con- 
suming. Neighborhood schools, however, are frequently within walking 
distance of the home. 

• Prevention. An advantage of working with children is the potential for 
problem prevention. Among programs that focus on prevention are 
WHO, offered by the Mental Health Association to prevent child abuse, 
and Project DARE, offered by the local police to prevent drug abuse. 
Both programs are delivered through the schools because of the efficien- 
cy with which children can be reached. The School of the Future con- 
cept incorporated both preventive and treatment services. 

Proposals and Goals 

After each site agreed on which schools would participate in the project, repre- 
sentatives from the sites were asked to read the document explaining the project’s 
concept. The Foundation then asked site representatives to submit proposals for 
frinding. Two of the sites had school personnel with considerable experience writ- 
ing grant proposals; two sites were not able to find such experienced staff. The 
Foundation was proactive in the proposal development phase, providing techni- 
cal assistance in this area ranging from brainstorming with site representatives 
about project goals to helping site representatives develop and write the proposal. 

O 



Foundation staff provided this assistance for two important reasons: 1) to 
become more knowledgeable about how the site planned to use the grant 
money (goals, objectives, budgets) and 2) to ensure that each site submitted a 
proposal. There was some concern that one or two of the sites might not find 
school staff with the time and expertise to develop a proposal. Since all four 
sites had already been approved informally by the district administrators, prin- 
cipals, and the Foundation, everything possible was done to make sure this 
opportunity was not missed. The final hurdle was to obtain formal approval 
for the grant from each site’s district school board. This was accomplished 
fairly easily and quickly after the grant proposals approved by HF were 
reviewed by the board members. 

Since the creation of an evaluation plan for the project was based on what the 
sites intended to accomplish over a five-year period, the objectives and goals 
developed in the proposals were of great importance to the evaluation team 
(Culler and Keir, 1992). Some general goals for the project were laid out in 
the concept paper shared with the sites, but each site was responsible for 
developing its own set of specific goals for the project. This flexibility, 
enabling each site to define its vision of the School of the Future, was one of 
the features of the project that made it appealing to school administrators. On 
the other hand, as will be explained later, it also made it much more difficult 
to develop a standardized across-site data collection plan. Following is a sum- 
mary of each site’s project goals as stated in the grant proposals received by the 
Hogg Foundation in the spring of 1990. 

Austin 

• Stabilize the community’s population. 

• Define the community better. 

• Provide more efficient and effective delivery of needed services to fam- 
ilies in the community. 

• Improve the functioning and performance of the students. 

Dallas 
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• Provide a concentration of district services at a cluster of schools serv- 
ing students and their families from kindergarten through eighth grade. 

• Establish problem-solving teams to address school and community prob- 
lems and the problems of individual students and their families. 

• Work with families in crisis to reduce stresses that inhibit their ability to 
support their children’s success in school. 

• Develop services designed to teach parenting skills. 

• Develop a cluster of community services and resources tailored to the 
needs of neighborhood families and coordinate and make them accessible 
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through the schools. 

• Evaluate the impact of the program through attendance and achieve- 
ment gains in students as well as by measures of improved motivation, self- 
esteem, behavior, increased parental involvement in school activities, and 
improved availability and utilization of community services. 

Houston 

• Improve the social and academic performance of the students by involv- 
ing their parents and the community in their education. 

• Identify and build upon the strengths of children, families, schools, and 
the community. 

• Offer enrichment programs that promote self-esteem and positive 
human development. 

• Coordinate services for children and their families in their own neigh- 
borhoods. 

• Prevent or treat a variety of problems such as substance abuse, child 
abuse, school dropout, teen pregnancy, and suicide among high-risk youth. 

San Antonio 

• Establish a school climate that is conducive to education and personal 
enhancement. 

• Bring support services to the school campus that are needed to improve 
student learning and mental health. 

• Develop a cooperative working relationship among different communi- 
ty agencies and persons who can provide supportive services to the school. 

• Obtain the types of support services needed on a voluntary basis when- 
ever possible. 

• Develop financial resources to enhance learning and to improve the 
well-being of children and families. 



DESIGNING THE EVALUATION 
The Strategy 

School-based or school-linked service projects tend to be complex, making it 
difficult to evaluate their effectiveness. Kehrer (1993) states, “The greatest 
range of options and the greatest potential for spending on evaluation, and 
hence the most difficult decisions about how to evaluate, arise with grants for 
direct services to people and grants for educational or training programs” (p. 
27). The School of the Future project, with its multiple sites and site-specif- 
ic program development, was no exception. There were very few evaluation 
models to follow. Several documents highlight this dilemma. The final report 
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of the Commission on the Mental Health of Adolescents and Young Adults 
stated that “evaluations of school-based service delivery efforts, based on 
sound data, are essential, (but) they are also scarce” (p. 15). Gomby and 
Larsen (1992) write that the evaluation of school-based services is “both live- 
ly and less than fully explored” (p. 68). Therefore, the Foundation thought it 
was important to organize a committee of evaluation advisors who would meet 
regularly to discuss issues and offer advice on methods for handling many of 
the difficult evaluation questions. 

This group of advisors, called the Evaluation Review Committee, was com- 
prised of four faculty members from The University of Texas at Austin 
(Departments of Psychology, Educational Psychology, and Social Work); three 
experts in evaluation (one for each site outside Austin); two representatives 
from the Texas Education Agency; and all of the program officers from the 
Hogg Foundation. 

Evaluation and research activities for all four sites were coordinated by a full- 
time director of evaluation at the Foundation. In addition, each site was 
assigned a half-time research associate working at the Foundation to help with 
data collection and analysis under the director’s supervision. The evaluation 
director worked closely with these research associates, the Evaluation Review 
Committee, and key personnel at the four sites to design the evaluation and 
oversee its successful implementation. 

The Goals 

The overarching goal of the School of the Future project was to enrich and 
enhance the lives of children and their families in their communities. 
Student education, physical health, and mental health were all expected to 
experience positive changes, over time, as a result of the project. To accom- 
plish this, the project also emphasized the development of a long-term col- 
laboration between local human service agencies, public school systems, and 
communities of teachers and parents. Some of the objectives associated 
with this goal were given a higher priority than others. The following are a 
few of the many areas in which the demonstration sites and the Hogg 
Foundation anticipated seeing an impact over the five-year period of the 
project. 

For the students: 

• Academic performance: improve attendance, reduce number of 

dropouts, improve school achievement. 

• Physical and mental health of students and families: increase immu- 
nizations, reduce substance use and abuse, lower teen-pregnancy rates. 



reduce physical abuse, reduce emotional/sexual abuse, improve self-moti- 
vation, improve self-esteem, lower stress levels, reduce discipline referrals, 
decrease gang activity. 

For the families: 

• Involvement between family members and their children: increase 
number of parenting education classes, increase number of consumer and 
life skills classes. 

• Involvement of parents with the schools: increase participation in PTA 
and partnership councils, increase participation in school-sponsored class- 
es and activities such as adult living skills and English language skills. 

For the schools: 

• Number of available and affordable neighborhood services: assess cur- 
rent level of available services and resources, assess immediate needs of 
community, increase number of affordable services offered, increase inter- 
agency coordination and the development of new funding sources, maxi- 
mize use of volunteer organizations. 

• Supportive school environment for students, teachers, parents, adminis- 
trators, and community partners: decrease teacher attrition rate, improve 
teacher attitudes, improve students’ perceptions of school, improve par- 
ents’ perception of children’s schools. 

For the communities: 

• Image of the school in the community: create a positive perception of 
the project, improve the perception of the school and the teachers, 
improve the perception of the neighborhood. 

• Student, parent, and teacher participation in the community: increase 
the use of services by community residents. 

• Integration of school and community activities: increase interaction 
and integration of community and school activities. 



Quantitative Measures 

During a visit to school-based service projects in New Haven, Connecticut, 
and Philadelphia, Pennsylvania, the SoF evaluation director learned several 
important lessons regarding the collection of evaluation data. First, data col- 
lection should emphasize the demonstration of what is and what is not effec- 
tive about the project. Second, this information must be shared with school 
staff on a regular basis to assist in their planning. Third, the school staff 
should not be overburdened during the data collection process. That is, staff 
should not be taken away from their responsibilities to the school. This infor- 
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mation was considered carefully before making decisions about which instru- 
ments were to be used and how data were to be collected at the sites. 

Decisions still needed to be made regarding the survey instruments to be used 
in the collection of quantitative data in each of the SoF sites. Members of the 
Evaluation Review Committee met several times with SoF evaluation staff 
and spent many hours reviewing various instruments and sharing their expe- 
riences with surveys designed for collecting data from children. 

The staff listed several potential mental health instruments and considered 
the efficacy of using these instruments. A summary of the strengths and weak- 
nesses of each instrument was shared with the committee (Witt, et al, 1990). 
Based on the results of this systematic review, the committee determined that 
Achenbach’s Youth Self-Report and Teacher Report Form were the best 
instruments of those investigated for the target population, primarily because 
they would enable the project to capture mental and physical health informa- 
tion on students through the perspectives of both the children and their 
teachers. Included with the self-report version of the survey were instruments 
exploring student self-esteem and student perceptions of the school climate. 

Teacher Report Form and Youth Self'Report Assessments. The Achenbach 
Teacher Report Form (TRF) is one of a series of three instruments developed 
to assess psychopathology in children. It was designed to be completed by the 
child’s teacher and contains 112 problem items, each of which the teacher 
rates as “very true,” “sometimes true,” or “never true” of the child’s behavior 
as observed over the previous six months.^ 

As standardized by Achenbach, there are eight first-order (narrow band) sub- 
scales derived from these problem items: Withdrawn, Somatic Complaints, 
Anxious/Depressed, Social Problems, Thought Problems, Attention Problems, 
Delinquent Behavior, and Aggressive Behavior. In addition, there are two sec- 
ond-order (broad band) subscales: Internalizing (Withdrawn, Somatic 
Complaints, and Anxious/Depressed combined) and Externalizing (Delinquent 
Behavior and Aggressive Behavior combined). A Total Problems scale also can 
be calculated which is the sum of all of the problem items in the survey. 

The TRF was developed on a nationally representative sample of children, 
and research by Achenbach and others has demonstrated evidence supporting 




^The Achenbach instrument also contains several “competence items" which were not col- 
lected for this project. 
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the reliability and validity of the instrument (Achenbach, 1991a). It was 
designed to be used along with two companion instruments: the Youth Self- 
Report (YSR), which is completed by children from ages 11 to 18, and the 
Child Behavior Checklist (CBCL) which is completed by the parents of those 
children. The CBCL was not used in this study. 

The YSR is almost identical to the TRF, differing only in that it asks the child, 
rather than the teacher, to respond to the survey. Like the TRF, the YSR con- 
tains 112 problem items which children rate using “very true,” “sometimes 
true,” or “never true” of thoughts, feelings, or behaviors they have experienced 
in the past six months.® 

Subscales derived from the items with the YSR are identical to the TRF sub- 
scales except for a Self-Destructive/Identity Problem subscale developed 
exclusively for boys. The two second-order subscales labeled Internalizing and 
Externalizing are scored identical to those in the TRF; the Total Problems 
scale is obtained by adding all of the problem items as in the TRF. The YSR 
instrument was developed on a nationally representative sample of children 
and, like the TRF, research using the YSR has demonstrated evidence sup- 
porting the reliability and validity of the instrument (Achenbach, 1991b). 

Self-Esteem. The self-esteem measure used in the first year of the survey was 
Harter’s Perceived Self-Competence Scale (Harter, 1982; 1985). Since the 
results of the initial analyses on this scale showed very low levels of reliabili- 
ty when testing the middle school students in all of the sites, the Evaluation 
Review Committee suggested that a different measure of self-esteem be sub- 
stituted for the Harter. Rosenberg’s Self-Esteem Survey was recommended 
and incorporated into the self-report administrations from the second year on 
with students in grades six through eight. Rosenberg’s Self-Esteem Survey 
offers statements asking students to respond with a four-point scale that ranges 
from “strongly agree” to “strongly disagree.” The survey contains ten items 
and takes fewer than five minutes for students to complete (Rosenberg, 1965). 
Tests of reliability using the Rosenberg Self-Esteem Survey resulted in signifi- 
cantly higher scores than did the Harter, thereby convincing evaluators that 
they should continue using this survey. 

School Climate. The National Education Longitudinal Study (School Life 
section) was used as a measure of students’ perceptions of school climate. 



®As was the case with the TRF surveys, the “competence items" were not collected in the YSR 
survey administrations. 
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Students in grades six through eight were asked to respond to 13 statements 
concerning how they feel about their school, their teachers, and their class- 
mates, using a four-point scale ranging from “strongly agree” to “strongly dis- 
agree.” The survey takes fewer than five minutes to complete and has been 
used extensively with students as part of a larger survey of school issues 
(National Center for Education Statistics, 1988). 

Ethnic Identity. A measure of ethnic identity also was incorporated into the 
self-report version of the survey. Of the 20 items used by Phinney (1992) to 
develop three ethnic- identity subscales in her research (affirmation and belong- 
ing, ethnic identity achievement, and ethnic behaviors), 6 of these items were 
included in the SoF survey packets. Two additional questions exploring this 
issue, but not related to Phinney’s scale, asked students about their use of the 
Spanish language at home with their parents and at school with their friends. 

Survey Concerns. The Evaluation Review Committee voiced several con- 
cerns about the survey instruments considered by the project and asked the 
SoF staff to respond to them. After investigating the issues raised by the com- 
mittee, the SoF staff met again with the committee to explain the practical 
and theoretical reasons for selecting the Achenbach YSR and TRF assessment 
tools. These reasons included the following; 

• Both the YSR and the TRF have established statistical norms that enabled 
the project to compare data collected from the SoF sample of children to a 
national sample of children. 

• The information collected by the TRF is nearly identical in content to the 
information collected by the YSR. Over time, this allows for a comparison 
of psychological profiles of students by teachers as well as profiles by students 
themselves. 

• The TRF allows for the collection of behavioral information on children 
who are too young to complete the YSR survey, enabling the project to 
develop syndrome scales similar to the YSR at lower age levels. 

• Both the YSR and the TRF have been tested rigorously on children of 
diverse backgrounds and culture, including children from countries outside 
the United States. Cross-cultural research using other instruments was not 
available. 

• The YSR was short enough to be administered in the time frame allotted 
by the schools. One class period, or approximately 50 to 60 minutes, was all 
the evaluation staff could expect to use of the students’ time each school year. 
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Methodology 

Target Population. Teachers of grades one through five were asked to rate all 
children in their current classes for whom parental consent was obtained. At 
the principals’ request, kindergarten classes were included in some of the sites. 
The TRF was conducted at the Austin, Houston, and San Antonio SoF 
schools (experimental) and demographically-matched schools (comparison) 
in each of the sites. One comparison school was selected for each SoF school 
participating in the quantitative evaluation.^ This quasi-experimental 
research design was used so evaluation staff could compare outcomes for stu- 
dents in schools receiving services to a similar group of students not receiving 
those services. 

All children in grades six through eight who had obtained parental consent 
were eligible to participate in the self-report study, whether or not they were 
receiving any services offered through the school. After some attrition due to 
the lack of parental consent or student absence due to illness, an average of 
over 80 percent of the students in these schools were assessed every survey 
period. While Hispanic youth prevailed as the dominant ethnic group in both 
the teacher report and self-report surveys, especially in San Antonio, there 
were somewhat larger proportions of African-American and Anglo youth in 
the Austin middle and elementary schools. 

Data Collection. One middle school and one or two of its “feeder” elementary 
schools, all serving families in the same neighborhood, participated in the pro- 
ject at each SoF demonstration site. The TRF and YSR were administered annu- 
ally over a three-year and four-year period, respectively, in the three sites partic- 
ipating in the quantitative evaluation — Austin, Houston, and San Antonio. 

The TRF surveys were completed by teachers at each of the three sites in 
March through May of 1992 through 1994. This gave the teachers a full six 
months to get to know their classes before completing TRF assessments on 
their students. The youth self-report, self-esteem, and school climate surveys 
were administered in the fall of each school year at the middle schools in the 
participating sites. 

Consent Procedures. Consent procedures for obtaining permission to collect 
ratings varied by site and school, based on the concerns of the superintendents 



^After the third year the decision was made to discontinue data collection at the comparison 
schools due to concerns regarding the cooperation of staff in these schools, the efficiency of 
using financial and human resources in this way, and the quality and utility of these data. As 
a result, data from the comparison schools are not included. 
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and principals. In some schools active consent was used. Active consent 
required that the child’s patent read, sign, and return the consent form before 
the child could become eligible to patticipate in the survey. Passive consent 
required only that the fotm be sent home, either with the student or by mail. 
If the parent did not want the child to participate in the survey, the parent had 
to sign and return the form with the “No” category checked. If the form was 
not returned, it was understood as tacit consent for the child to participate. 
This procedure was spelled out clearly on the consent form, and forms were 
written in easy-to-read language in both English and Spanish. (See Appendix 
B for an example of the active and passive parental consent forms.) 

For each site, stamped, self-addressed envelopes containing completed forms 
were returned to the Hogg Foundation. Students were provided with incen- 
tives for returning these forms promptly. Names, student IDs, and consent sta- 
tus were entered into a computer database which was updated frequently. 

For the self-report surveys, middle school students with parental approval for 
inclusion in this survey were also asked for their permission to participate at the 
time of survey administration. (See Appendix C for a sample of the child 
assent form used in the self-report surveys.) For the teacher report surveys, 
teachers interested in assessing elementary school students who had an affir- 
mative consent form on file were also asked to complete consent forms. (See 
Appendix D for a sample of the teacher consent form used in the teacher report 
surveys.) Teachers were paid for each child on whom they completed a TRF.*° 

San Antonio chose to use the active consent procedure. Representatives at 
the site believed that many parents in the community probably would not 
return the consent forms if they were mailed or sent home with students. 
Since this would lead to a severe reduction in the number of students eligible 
to participate in the survey, paid workers and volunteers made home visits in 
the first year to collect signatures from parents in the community. The con- 
sent form authorized the evaluation staff to collect data for all children with 
signed forms over the entire five-year period of the project. Although this 
method was far more costly and time-consuming than passive consent or 
active consent by mail, it enabled the project in San Antonio to collect data 
on many more students than would have been possible otherwise and also pro- 



’°Teachers were paid $100 per classroom the first year, but when teachers discovered that 
some were assessing 25 children while others were assessing 10 children for the same 
amount of money, a more equitable method was implemented the next year — $5 for every 
completed survey. 
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vided an opportunity to inform individual parents about the SoF project and 
the services available. 

In Austin, active consent was used in the comparison school, but passive con- 
sent was chosen for the experimental schools. Forms were mailed to parents 
using a mailing list compiled by the school district. In Flouston, passive con- 
sent was used in all the schools except for special education classrooms in the 
comparison site, where active consent was required. 

Incentives for Participation. Teachers and schools were provided with finan- 
cial incentives for participating in the collection of data. Incentives were 
considered important because teachers and school staff, already extremely 
busy, had to find extra time to complete these assessments and were reluctant 
to devote even one period of instructional time each year for participation in 
the self-report surveys. For the YSR, every participating classroom was award- 
ed $25 to use in any way the teacher deemed appropriate. The payment to 
teachers and the financial contribution given to the classrooms resulted in a 
high rate of participation for students and teachers in these sites. 

Incentives also were used to facilitate the consent process. It was difficult 
to get students to return signed consent forms, regardless of whether the par- 
ent responded “Yes” or “No.” To remedy this, students were offered a reward 
for returning consent forms within a given period of time, whatever the par- 
ent’s response. The particular incentive to be used was discussed with the 
principal and teachers and in all cases candy, as opposed to pencils or school 
supplies, was the item of choice. Evaluators and school staff agreed that 
these incentives resulted in a greater return rate than would have resulted 
without them. 

Qualitative Measures 

While it was determined that outcomes for children and families were important 
areas to monitor at regular intervals, a process evaluation of the project also was 
an integral part of the research design. Keeping track of the project’s develop- 
ment, the various responsibilities and activities of the project coordinator, and 
participating families was considered important because this information could 
validate the project’s effectiveness to potential funders. It could also prove help- 
ful to other school districts interested in developing school-based service projects. 



"Since resources were not available for home visits in Houston, few parents returned the 
forms in the special education classrooms, resulting in special education students being 
dropped from the comparison school sample. 
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For the qualitative evaluation, SoF staff and the Evaluation Review 
Committee developed two interview strategies. One was to be conducted 
twice with key representatives at the sites, once near the beginning and again 
near the end of the project. The other was a one-time interview to be con- 
ducted with a sample of families at the sites in the project’s final year. 

Key Informant Survey. The Key Informant Survey (KIS) focused on: a) pro- 
ject strengths, weaknesses, and barriers, b) perceptions of the effectiveness of 
the project coordinator and the project overall, and c) opportunities for insti- 
tutionalization of the project. Interviews were conducted with key individu- 
als in each site early in 1991 and again, when possible, near the end of the 
demonstration period. Also interviewed in year five were other key persons 
who became familiar with the project after the first year. 

Research associates from the Hogg Foundation were trained in interviewing 
techniques to conduct the KIS. An average of 15 individuals was interviewed 
at each site, including all project coordinators and school principals and a 
sample of counselors, teachers, district administrators, service providers, and 
parents. Those interviewed represented a cross-section of participants at each 
of the demonstration schools and their communities. Each interview took 
from 60 to 90 minutes. 



Project Coordirmtor Interview. At the beginning of the project, the evalua- 
tion staff asked the coordinators to keep a log or journal of their daily routine 
including tasks occupying most of their day, time spent on administrative and 
service responsibilities, job skills used most often, and skills considered most 
important to the successful completion of their job. However, because of the 
coordinators’ busy schedules, keeping a journal was simply not possible. 
Therefore, the evaluation team decided to initiate regular interviews with the 
coordinators to collect this information. 



A senior research associate from the evaluation staff interviewed the project 
coordinator — sometimes in person, other times by phone — at all four of the 
demonstration sites every three or four months. Although project coordina- 
tors were the targeted respondents for these interviews, others at the sites who 
were interviewed periodically included principals, teachers, parents, district 
administrators, and key public and private partners associated with the pro- 
ject. The information collected from these interviews was used to develop a 
more detailed picture of the project coordinator position, which was consid- 
ered the unique aspect of the project. Information from these interviews was 
incorporated in a booklet that uses case examples to describe the coordinator 
position and demonstrate its importance in the SoF project (Iscoe, 1995). 
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Family Interview. An in-depth family interview was developed to capture a 
variety of information from the parents of students participating in the School 
of the Future project. The interviewers, all of whom were members of their 
respective site communities and were trained in interview techniques by SoF 
evaluation staff, asked questions regarding assistance received from the school 
and other service agencies, parent involvement in the school and their chil- 
dren’s education, and behavior of their children at school and home. They 
collected information on the parents’ familiarity with the SoF project at their 
children’s schools as well as basic demographic information. Interviewers also 
administered a “life events’’ checklist on extreme events or crises that had 
occurred within the six months prior to the interview. Finally, they were 
asked to observe and keep notes on general conditions in the homes and any 
interaction they may have witnessed between parents and children. 

Each interviewer was given a list of phone numbers and addresses for parents 
of students for whom two or three years of mental health information had 
already been collected via the Achenbach YSR and/or TRF This list includ- 
ed names of students attending the SoF schools who had received services 
from the project as well as those who had never received services from the pro- 
ject. Since multiple-year surveys were available for over half of the students 
surveyed, a fairly large sample of families existed from which to select. 

Interviewers contacted the parents of the students listed by telephone and 
scheduled appointments to meet in their homes. Each family participating in 
the survey received a $15 gift certificate from a local grocery store, while inter- 
viewers were paid $30 for each interview conducted. The survey lasted 
between one-and-a-half to two hours. 




27 



PART II 



chulleikes encduiitered hd lessons lenrned 



I n Full Service Schools, Dryfoos (1994) states that “we must start off with a 
strong caveat: The state of the art of program evaluation leaves much to be 
desired” with regard to multisite projects such as comprehensive school- 
based programs (p. 123). Dryfoos could identify only three such programs 
that generated outcome-based evaluation findings. Her concern was reinforced 
by the evaluation staff of the School of the Future (SoF) project (Millea, 1996). 

One unique aspect of the SoF project was the Foundation’s commitment to a 
longitudinal evaluation. This section focuses on the lessons learned from the 
evaluation. It is not an attempt to define an “appropriate” research design. 
Rather, it describes our experiences and what we learned from these experi- 
ences. 




FACING DIFFICULTIES 



Project Expectations 

Professionals who have worked in the field of human services research are well 
aware of the difficulties of evaluating complex social programs (Kehrer, 1993). 
Since the School of the Future project was initiated in the summer of 1990, 
more literature on the evaluation of comprehensive service programs has 
emerged, including the effects of such programs on the processes of evaluation 
(Brown, 1995). But there are other difficulties. As Dryfoos (1994) indicates, 
school-based health and social service programs seem to have been held to a 
more rigorous standard of success than other social programs, with expecta- 
tions that they can rapidly alter student performance and behavior as well as 
provide other benefits. Perhaps these expectations exist because achieve- 
ment data (e.g., grades and standardized test scores) are readily available from 
school districts and often used in policy-making. 



Partners and Priorities 

Complex, collaborative, community projects are likely to have multiple goals. 
These may vary over time and may even be conflicting. Examples of such 
goals encountered by the SoF project included documenting needs, influenc- 
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ing policy makers, improving student academic achievement, stabilizing fam- 
ilies, reducing attrition, increasing community activism, improving the health 
of children, and testing the effectiveness of a particular intervention strategy. 

Having multiple goals means that there will be several research questions 
about which information must be collected. Because many programs may 
require collaboration, some goals and research questions may be of greater 
importance than others to some partners, creating different expectations. 
This difference can lead to conflict among partners. An example encountered 
by the SoF project was the funding of a school-based health center by one site. 
When the evaluation of this service was being developed, two research ques- 
tions were considered. Should the team look at; 1 ) improvements in student 
health as measured by reduced absenteeism or 2) reductions in the use of 
emergency room care for illnesses by the students’ families at the city-owned 
public hospital? 

From a fiscal standpoint, and since the city was a key partner, it made sense to 
evaluate the second question. But in this case the first question was of greater 
importance to the school district. Resources were unavailable to address both 
questions. The research issue was resolved pragmatically in this case. Primary 
care services were not available through the school health center. The school 
board had restricted care at the school-based health center to preventive ser- 
vices, thus limiting impact on emergency use. Therefore, evaluation focused 
on the first question, which was revised to follow the impact of services on 
those children identified as “chronically absent” rather than all students in the 
school. This change helped reduce the possibility that a severe outbreak of an 
illness would negatively impact the results. These were more difficult data to 
obtain, but when these data were analyzed, the results of the analyses were a 
better fit for the question asked. 



Partnerships for Evaluation 

Defining and achieving agreement on the goals and research questions 
through collaboration is essential to developing a good evaluation plan. 
Failure to do so results in wasting valuable resources and an inability to cap- 
ture the relevant information. 



One way to deal with potential conflicts over the focus of evaluation is to 
have partners independently fund some aspects of the evaluation pertinent to 
their own missions and goals, while collaborating in other areas. This further 
extends resources for the partnership. A willingness to share data among part- 
ners, and the ability to share data files while respecting the need for client 
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confidentiality, are other important considerations for developing an effective 
and efficient evaluation. Those interested in designing an evaluation plan 
might seek references on the processes of collaboration or recruit a know!' 
edgeable group facilitator to guide the group through this process. 

Lesson Learned ... In complex collaborative projects it is crucial to 
reach some early agreement among partners regarding the specific 
goals and research questions for evaluation. Failure to do this can 
result in a conclusion that the project has accomplished little. 

Student Variables 

Improvement in student attendance, grades, and standardized test scores and 
decreases in student disciplinary incidents are very often desired outcomes of 
school-based social service interventions, but their use as the sole outcome 
measures of a project can make a successful intervention program appear unsuc- 
cessful. For example, comprehensive school-based programs that make 
progress in preventing school dropout will often increase the attendance rate 
for at-risk students who were likely to have dropped out of school without 
intervention. This success, in turn, can result in an increase in student disci- 
plinary problems and a decrease in student achievement when measured at the 
school level, at least in the short term. The reason for this is that at-risk stu- 
dents who formerly dropped out of school now are in attendance, and therefore 
they are included in measures of student performance. Thus, it is often the case 
that performance in school-wide academics declines before it improves. 

The current wave of research on school-based programs indicates that children 
whose families receive extensive comprehensive services will most likely be the 
ones to show the most changes early on, largely because their needs are more 
extreme (Golan &. Wagner, 1996). However, because most families targeted for a 
community do not experience such extreme need, the impact of an intervention 
is likely to be less evident and take longer to show significant positive change. 

Lesson Learned . . . An evaluation designed around collecting data 
merely because certain information is available can result in misleading 
conclusions regarding the long-term effectiveness of a service or project. 



Population Mobility 

Another difficulty with conducting research which focuses on disadvantaged 
families is the mobility of the population. In a setting where as much as half 
of a school’s enrollment may change during the course of the school year, 
tracking students over time can be complex and expensive. Even if students 
can be tracked, there is a good chance that they will not remain in the school 
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long enough to benefit significantly from the services offered. In one SoF site, 
although the yearly survey of middle school students was completed by about 
90 percent of the students in the school, only 15 percent of those students 
could be tracked over all three middle school years. This type of attrition can 
negatively impact the research due to the reduction in sample size and result- 
ing loss of statistical power. 

Lesson Learned ... It is important to factor demographic changes over 
time into the sampling methods. Substantial changes in mobility and 
migration can seriously affect a target sample and severely limit the 
generalizability of the results of the study to a larger population. 



Selection of Measurements 

It is extremely important to match measurements to outcome goals. In the 
SoF project the main intervention was systemic — the position of project coor- 
dinator was introduced to secure and coordinate services that would use the 
school as the locus for service delivery. The SoF evaluation collected infor- 
mation on the impact of the project process as well as data on the outcomes 
for students over time. 

In one SoF site, for example, student mobility was approximately 50 percent. 
There, improving community stabilization was perceived as a precursor goal 
for improving student achievement because families needed to remain in the 
community long enough to receive the interventions offered if an impact on 
student performance were to be achieved. Student mobility, therefore, was a 
primary goal at this site and an important measure to consider when explor- 
ing the project’s impact on school performance. 

Lesson Learned . . . Clearly identify the goals of the intervention and 
develop variables to measure those goals and objectives accordingly. 
The evaluation design must relate closely to the kinds of the inter' 
ventions developed as well as the accompanying theory of the design. 

Targeting individuals who receive the available services is another important 
consideration in the evaluation design. SRI International’s statewide evalua- 
tion of a school-linked services initiative in California found that even when 
services were effectively delivered, these services were not usually related to 
school-wide improvement. The authors of this report state that “a balanced 
continuum of services that ranged from prevention-oriented services to more 
intensive services was associated with improvement in student behavior and 
performance” (Golan and Wagner, 1996, p. 5-1). 
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Since only a small proportion of students involved in school-based programs 
is likely to receive intensive services, the use of school-level data solely to 
evaluate program effectiveness most likely will not be helpful in identifying 
whether a program is having a positive impact on students. Students receiv- 
ing services must be identified and the frequency with which they receive 
these services indicated over the course of the project. This population is the 
appropriate focus of analysis for determining the impact of services on indi- 
vidual student performance.^^ 

Lesson Learned . . . Consider the level at which change is measured. 

A strategy must he developed that identifies students receiving services 

and tracks the frequency with which they receive them. This is the 

only way to accurately measure the effectiveness of a given service. 

Intervention and Outcome Links 

Yet another difficulty in conducting evaluation research of programs is mak- 
ing a direct link between an intervention and the impact that intervention 
has had on those participating in the program. Because both the problems and 
the interventions designed to address them are complex and multidimension- 
al, it can be difficult to determine whether a particular intervention was 
responsible for achieving a desired outcome. 

For instance, in one site it was suggested that the provision of comprehensive 
school-based services through the SoF was the major factor in slowing the rate 
of student attrition at the elementary school. The principal believed that par- 
ents were remaining in the neighborhood because social services were being 
offered through the school.^^ A city official challenged this statement, indi- 
cating that improvement in the local economy was responsible for a housing 
shortage which, in turn, resulted in parents remaining in the neighborhood 
longer. With occupancy rates up and rent costs increased around the city, 
families no longer could afford to move easily. 




^ ^Focusing on these smaller groups of students in an effectiveness study can create other 
design problems which require attention. Potential problems to be aware of include, but are 
not limited to, sample size, missing data, and attrition rates. They can impede the analysis 
through decreased statistical power. These problems, however, are quite common in social 
services research. 

^^Information collected through the Key Informant Surveys supported this principal's expla- 
nation. 



^"^It could also have been argued that rising rents in this community resulted in rent-skipping 
behavior, family destabilization, and increased student mobility at school. 
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The reduction in mobility which was observed could more arguably be tied to 
both the services available through the school and an improving economy, as 
well as such other factors as the emergence of neighborhood leaders and 
activists running out drug dealers. It is difficult to sort out what causes any 
resultant change when so many other variables are part of the equation. In 
fact, this example demonstrates how data can be used to support contradicto- 
ry theories. As a result, one must be extremely cautious when identifying the 
causes of change in a project over time. 



RECOGNIZING BENEFITS 
Unanticipated Benefits 

The School of the Future experience demonstrates that integrating evaluation 
with service delivery is important for the survival of a project. One reason is 
that, given the nature of human services funding, it is unlikely that programs 
that fail to include an evaluation component will receive funding support. 
Further, the nature of the evaluation component needs to be more than sim- 
ply recording utilization rates such as tracking the number of people served by 
a particular service. 

The type and timing of evaluation activities can be used to serve the goals of 
program development and implementation, but they can also have additional 
advantages. For example, a needs assessment at one site, located in a new 
neighborhood in which few residents knew one another, became a tool for 
reaching out to parents, helping them get acquainted, and drawing them into 
the project. Training and involving parents in the data collection process pro- 
vided other benefits. By helping with data collection — going from door to 
door to meet their neighbors and interview them about school and communi- 
ty issues — neighborhood residents learned about the importance of needs 
assessments and baseline information in approaching local government for 
services. From this initial participation, a few community members emerged 
as local leaders, gaining skills in public speaking and advocacy and ultimately 
establishing a positive relationship with city officials. In large part because of 
their ability to document their needs in presenting their case, more than $1 
million of local funds were directed to the community. 

This neighborhood’s efforts were cited as a model by city staff when the 
funding environment tightened, because members knew how to collect data 
and interpret the results in a practical and meaningful way. One of the last- 
ing benefits of the SoF at this site was that key community members devel- 
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oped the skills to conduct and analyze surveys which collected input from 
the community. In this way they became less dependent on outside techni' 
cal assistance and better able to use their own skills for gathering and assess- 
ing information. 

Lesson Learned . . . Unanticipated benefits obtained from conduct- 
ing evaluation can include increased community awareness and par- 
ticipation, improved community empowerment and collaboration, 
and continued community development. 

Usefulness of Research 

Professionals engaged in service provision often find it difficult to devote time, 
effort, and resources to designing an evaluation plan when there are so many 
needs to be met. The provision of high-quality services for children and their 
families can be facilitated by a high-quality evaluation, but it requires a strong 
effort by service providers from the beginning of the project. It is easier for 
practitioners to make and keep a commitment to ongoing evaluation if the 
research design is sensitive to the needs of the population to be served by the 
project. 

The School of the Future was unique in that the Foundation’s financial com- 
mitment to the evaluation was comparable to the funds it provided for the 
demonstration grants. This enabled SoF researchers to consider a variety of 
approaches to evaluation and to benefit from their experiences, in turn 
enabling them to make practical recommendations to programs that might 
lack the resources and/or skills to evaluate a project. By sharing evaluation 
experiences, the SoF sought to improve the quality of evaluation and the use- 
fulness of the results as well as save others from reinventing the wheel each 
time an evaluation plan is designed. 

Service projects are encouraged to include evaluation and technical assistance 
resources in their proposals to funding sources. It is important to integrate 
methodological rigor into research design that will enhance the generalizabil- 
ity of results. 

Lesson Learned ... By sharing their results, insights, and experi- 
ences, project participants can help other programs as well as add to 
the knowledge base of evaluation strategies. 
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GRAPPLING WITH ISSUES 

Once the goals and research questions have been defined, developing a practi- 
cal and useful evaluation requires discussing and resolving a number of issues in 
three areas: the methodology to be implemented, the information to be col- 
lected, and the expectations for the evaluation. Resolving these issues success- 
fully is the key to obtaining results that can guide the project at the grass-roots 
level. An exploration of these issues, based on the SoF experience, follows. 



Methodology 

Process Evaluation. A process evaluation is crucial to the evaluation of any 
project. A formative evaluation can help identify whether the project is on 
target and identify problems early so that they can be corrected while the pro- 
ject is in operation. This is particularly important for a new effort. Tracking 
the progress of a project over time can provide documentation that will help 
explain how an initiative evolved. This is especially important in communi- 
ty development projects that include empowerment or leadership develop- 
ment as goals. Collecting process information may be the key to identifying 
leaders at the site as they emerge and gain community support. 

This type of information can be collected by using brief reporting forms con- 
sistently at all community meetings, planning sessions, etc.*^ These forms can 
be used to help identify issues and collaborative problem-solving processes 
that can help later with decisions regarding the achievement of desired out- 
comes. Consistency in documenting events over time is the key. Paper or an 
automated system can be used to capture this information. 

External Events. In addition to tracking internal processes, insight can be 
gained by a simple, low-cost method of tracking community or city events that 
are relevant to the project. Responsibility for the consistent documentation 
of events should be assigned at the beginning of the project. The SoF project 
documented school site, school district, neighborhood/community, city, state, 
and national events each school year. A word-processing table was used to 
collect data, which amounted to one or two pages per year. Collecting data as 
it occurs can strengthen the reliability of the information since it reduces the 
inaccuracies that can occur when relying on recall. 



’^These types of process evaluation collection forms are currently under development 
by Human Service Technology Innovations. 
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Outcome Measures. Tracking alone is not likely to be sufficient for most pro- 
jects. In the current funding environment there is a strong push for outcome- 
based evaluation. A smart planner will look for ways to measure the program’s 
impact on the persons targeted for services. Several resources provide practi- 
cal guides for developing outcome-based evaluation criteria, from tracking 
inputs and outputs (services provided) to assessing their effects on persons 
actually receiving services (Schalock, 1995; United Way of America, 1996). 

The most fundamental measure is the effectiveness study, which goes 
beyond counting the number of persons served to assess the ways in which 
the project affected the persons who actually received services. For exam- 
ple, did a child with a record of tardiness begin coming to school on time? 
Measures of impact, such as how the children were affected, cost-benefit, 
and the cost of accomplishing these changes, build on an effectiveness study 
and are important for seeking additional sources of funds to support a pro- 
ject. (Schalock, 1995). 

Lesson Learned . . . Both process evaluation and outcome-based 
measures of effectiveness are needed if a project is to contribute to 
the body of knowledge about collaborative community initiatives as 
well as secure continued financial support. 

Evaluation Measures. Though a formal effectiveness study should focus on 
those receiving the services, some measures of effectiveness can have multiple 
uses. For instance, the SoF survey of student perception of school climate was 
of considerable interest to administrators and teachers, identifying areas of 
positive and negative feedback and assisting them in understanding the con- 
cerns of students in their school. This survey also helped target emerging 
issues early. Similarly, yearly assessment of teacher perceptions could help 
identify and address morale issues, while periodic surveys of parents and stu- 
dents could help document the perception of needs and guide decisions 
regarding program initiatives. 



Lesson Learned . . . Resources can be leveraged by selecting mea- 
sures which serve multiple purposes. 



Impact of Services. Data tracking systems and evaluation instruments should 
include sources of information on the school and local community. There is 
a need for balance between the targeted study of effectiveness and other indi- 
cators of impact. Information on these issues that can serve as indicators may 
be readily available. Examples are changes in attendance patterns, school- 
level student achievement measures, changes in disciplinary actions, and 
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changes in juvenile crime statistics for the community. Local news articles 
also may provide information pertinent to the target community. 

Collecting Baseline Data. It is impossible to know if change occurred, or 
whether change has any link to the interventions provided, without collect- 
ing data at the beginning of the project. Data should include practical infor- 
mation such as role status, type, and frequency of interventions received by 
individuals and the costs of the various services provided. When this kind of 
information can be gathered accurately on individuals, the study is strength- 
ened. For the SoF, a simple tool for collecting parent participation informa- 
tion over time was a school’s sign-in book that was maintained for building- 
security purposes. By adding several columns to indicate the purpose of the 
visit and the role of the visitor, parent participation data could be collected 
throughout the school year. 

Lesson Learned . . • Resources can be utilized more efficiently by 
using readily available data sources such as news articles, published 
statistics, and school tracking forms. 



Survey Instruments. One way to begin an evaluation is to review validated 
instruments that address topics pertinent to the questions posed in the study. 
It is possible to avoid the fees associated with copyrighted instruments by 
using surveys available through the public domain. An alternative to paying 
for survey instruments is to develop your own. This can be time-consuming 
and will only offer face validity, which means that the results of the analyses 
can be generalized only to a population that is very similar to the one surveyed 
in your study. Depending on the purpose of the evaluation, this may be an 
acceptable limitation. 

Lesson Learned . . . Select data collection instruments based on the 
project's purpose and resource availability. Use public domain 
instruments or develop your own to meet budget constraints. 



Comparison Qroups. Because evaluation usually involves some comparison 
criteria, there is often a need to compare persons in a particular group with 
persons from a demographically similar group. The only difference between 
the two groups should be that one receives a service and the other does not. 
The important goal is to reduce all known sources of bias in the sample before 
evaluating whether a service is effective. 



One way to obtain a representative sample would be to select the entire pop- 
ulation — ^for example, all students in the school — for a study. If a high return 
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rate on the surveys can be assured, this could be a practical option. An alter- 
native is to select participants randomly, then assign half the participants to a 
group which receives the treatment and half to a group that does not. This is 
not recommended for social programs, however, since the research team 
should not deny services to those who need them just to ensure a clean 
research design for the evaluation. Moreover, it is highly unlikely the purity 
of such experimental and control groups could be sustained in any practical 
setting even if they were judged to be ethically acceptable. 

Another alternative is to use a waiting list of persons seeking access to a par- 
ticular program. Persons receiving services (experimental group) can be com- 
pared to persons in the waiting list group (control or comparison group). Still 
another practical option available in a school setting is to develop cohorts of 
students matching a predetermined profile who receive a service to demo- 
graphically-similar students who do not receive that service.'^ 

It is important to be consider what variables distinguish members of one group 
from members of the other, as any differences beyond the intended interven- 
tion may become a source of bias in the analyses. For instance, are students 
receiving treatment in one group because they got their permission slips in on 
time? Does this make them significantly different from the comparison group 
of students who failed to rum them in? These issues must be resolved before 
beginning the data collection process. 

When using a comparison group is not possible (for instance, when the num- 
ber of participants is too small or when developing a comparison group would 
mean denying services believed to be beneficial to a group), an alternative 
method is to study the same individuals over several points in time. This 
approach, called single subject methodology, uses the individual subject as its 
own comparison (Rubin and Babbie, 1989). The approach requires taking 
repeated measures of performance prior to any intervention (baseline period), 
and again as the intervention is introduced and periodically withdrawn. If the 
performance varies accordingly as the intervention is introduced and with- 
drawn, an argument can be made that the intervention is most likely influ- 



*^This method was implemented in the evaluation of the School of the Futute but was 
discontinued because it was not productive. Thete wete too many unconttolled, out- 
side influences in the comparison schools, such as the availability of alternative ser- 
vices similar to those received in the experimental schools. Such confounding of the 
key services being evaluated renders any conclusions involving the comparison schools 
questionable. 
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encing the behavior. Validity is enhanced as the process is repeated with 
other individuals experiencing similar outcomes. This methodology can be 
particularly useful with specific types of mental health interventions (e.g., par- 
ticular counseling approaches). It does, however, raise some methodological 
issues, particularly regarding generalizability of results. 

Parental Consent. The provision of comprehensive social services goes 
beyond the mission of education as it is understood by many people, and 
schools involved in comprehensive service provision may have had little 
experience in human service research and evaluation. The evaluation of such 
programs requires allowing parents to determine whether they want their chil- 
dren to have access to services and whether they will permit their children to 
be included in the service evaluation. It could be viewed as a good sign for 
the project when some parents refuse to have their children participate as it 
indicates that the informed consent process is working, parents are becoming 
empowered, and they are participating in their children’s education. 

Incentives. Incentives may seem like a luxury to include in an evaluation 
budget, but they are virtually a necessity when attempting to survey children, 
their parents, and their teachers. Incentives are effective in encouraging chil- 
dren and families to return materials, especially signed consent forms. This 
was jokingly referred to in the SoF project as the “Snickers Effect,” since the 
use of Snickers candy bars was so successful in getting students to return con- 
sent materials. In the SoF, consent materials were designed so that children 
received the incentive whether the parent accepted or declined the opportu- 
nity, as long as the form was returned. 

Similarly, in conducting surveys with middle-school students, teachers may 
experience difficulty managing the behavior of students who finish early or do 
not participate in the survey. To handle this situation, teachers in the SoF 
study initiated the idea of a coupon system, with one coupon redeemable at 
lunch for each student who met the behavior standards set by the teacher. 
The coupon could then be used whether or not students actually participated 
in the survey. These considerations might seem trivial, but the teachers con- 
sidered them both necessary and effective. The willingness to be flexible and 
creative helps gain the trust of teachers and insures their support despite dis- 
ruptions to the normal classroom schedule. 



'^There is extensive literature on this approach, in particular Rubin and Babbie (1989, 
p. 266-307) address this methodology in one chapter of their book. 
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Money should be set aside in the evaluation budget for another incentive: pay- 
ing key informants for their participation in the collection of appropriate data. 
Elementary school teachers participating in the SoF evaluation were paid $5 
for each survey completed on their students. Surveys, which took about twen- 
ty minutes to complete, were filled out by teachers on their own time. At the 
middle school, students completed self-report survey instruments under teacher 
supervision during a single class period (about 50 minutes) once a year for four 
years. A stipend of $25 was awarded to each classroom for its participation. 
The money was given to the school but earmarked for use in the classroom, 
with the teachers and students deciding how they would use it. Comments 
indicated that the teachers appreciated not only the money but also the recog- 
nition that their time was valuable to the school and the project. 

The SoF family survey also included an incentive: a $15 gift certificate to a 
local grocery store for each participant. Incentives can consume a consider- 
able portion of an evaluation budget, but they can be vitally important in the 
successful collection of evaluation data. 

Lesson Learned . . . The use of incentives for participation can con- 
sume significant program resources but also convey a level of respect 
to participants and facilitate the gathering of evaluation data. 

Information Collection 

School-Site Data. Data collected and stored at the school site are likely to be 
of better quality than data collected by the school district. It calls for organiz- 
ing some internal data collection processes, preferably easy-to-use systems that 
will not overload staff with additional paperwork. For example, school staff can 
record all the reasons students are sent to the office rather than just the major 
disciplinary actions required by the district, or identify the reasons people visit 
the school, not just who visits. Much can be accomplished by converting infor- 
mal record-keeping to more formal tracking systems. 

The most efficient systems for tracking and evaluation purposes are computer- 
based. Though some reasonably affordable and easy-to-use systems are now 
available, particularly for school-based health care, it is likely that more finan- 
cial resources will be needed for further development. In lieu of a computer- 
ized system, paper forms which can be converted readily to a spreadsheet for 
analysis may be an economic alternative. 

Precursor Measures of Academic Success. The use of standardized test 
scores may not, in the short run, yield productive results regarding program 



impact. In the earliest stages of the evaluation, precursor variables that affect 
long-term measures such as test scores should be the focus. Examples of pre- 
cursor variables are school attendance and disciplinary reports. Minor disci- 
plinary actions such as office referrals, which occur with frequency during the 
school year, might be better indicators than major disciplinary actions, such as 
suspensions and expulsions, which do not occur often enough to show change 
over time. When they are tracked for all students at the individual level, it is 
possible to determine if students actually receiving services are experiencing 
an increase or decrease in such incidents. Of course, a baseline measure must 
be established for the student before any change can be analyzed. 

Lesson Learned . . . The most useful data for the SoF project were 
obtained from individual student tracking at the school, using stu- 
dent identification numbers to link data. The district-generated 
identification numbers also helped protect the confidentiality of the 
students participating in the study. 

Data Sharing. High quality evaluation of comprehensive service programs 
relies on the ability to share data across different bureaucracies — health, social 
services, and education — as well as different computer systems. This requires 
the use of unique identifiers which enable one to evaluate student achieve- 
ment and physical and mental health measures while protecting individual 
student privacy. This is a challenging issue. In the SoF experience, the most 
reliable identifier was the school district’s student ID numbers, which proved 
preferable to the use of social security or service agency client ID numbers. 

A coding table can be used to correlate student ID numbers with other agen- 
cies’ client ID numbers to further protect confidentiality, at the same time 
allowing agencies to maintain their own systems of client identification. This 
table can also be a means of tracking individuals served by multiple organiza- 
tions across many variables pertinent to the comprehensive services effort. 
Further, it can reduce redundancies in data collection. The primary barriers 
to consider are concerns about confidentiality, turfism, and lack of technical 
expertise. Resources are currently being developed to assist service providers 
in addressing these issues (Halfours, et al, 1996). 

Consultants. Although it involves an additional expense to the project, hir- 
ing a consultant on a limited basis may be worthwhile to facilitate the data 
collection process and assure that information is gathered in a consistent 
manner and recorded accurately and promptly. To insure that the data are 
being entered accurately, a monitoring process may be needed until staff get 
used to the routine. If the routine is unobtrusive and beneficial to the school, 
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teachers, and parents, there will be better compliance. 

Feedback to the Sites. Students, teachers, parents, and community members 
are more willing to continue participating in research efforts when informa- 
tion is shared with them. Representatives at the SoF demonstration sites told 
interviewers that they were interested in learning more about their communi- 
ties as well as hearing information that could be used to help their projects. 
Presentation of information must be tailored to the targeted audience. 
Providing feedback gave the SoF a measure of accountability to the partici- 
pants and helped build trust for the project in the community. 

Feedback also can help explain why some expectations are not met. 
Comprehensive service programs have been placed under exceptional pressure 
to demonstrate improvements in student achievement, for example, but this 
type of change takes time. It is important for everyone involved to understand 
the complexity of the environment in which children live and learn and to 
recognize that changes in variables such as student achievement and mental 
health can take many years (Comer, 1980). 

It is the responsibility of the research staff to inform and educate parents, 
teachers, administrators, funding sources, policy makers, and legislators about 
the timelines involved in creating large- or small-scale change. 

Lesson Learned . . . One major use of evaluation data is to validate 
a program’s value when seeking funds for program continuation 
and/or expansion; another is to provide feedback to program directors 
and participants to help them improve services as well as encourage 
them to maintain their efforts. Although project evaluations are 
costly in terms of time, effort, and funds, their potential value should 
not be underestimated. 
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?hic Profile of Students Attending the Elementary School Sites: 1990-91 


San Antonio | 




51.0% 


49.0% 1 


100.0% 




98.0% 


0.5% 


1.3% 


0.2% 


100.0% 




23.2% 


18.0% 1 


17.1% 


15.1% 


13.1% 


13.5% 1 


o 

(Z> 


100.0% 




98.9% 








2 


(N 


00 

o 


(N 

CO 

00 




m 

i-H 

00 




i-H 

i-H 


CN 


CN 

CO 

00 




CO 

OS 

I-H 


o 

m 

I-H 


CN 

TJH 

I-H 


SO 

CN 

I-H 


(Tv 

O 

I-H 


CN 

I-H 

I-H 


o 


CN 

CO 

00 




CO 

CN 

00 








Houston ** 


S5 


49.7% 


50.3% 


100.0% 




82.8% 


14.3% 


2.8% 


0.1% 


100.0% 




22.3% 


18.5% 


14.9% 


1 15.47o 


14.3% 


14.6% 


o 

cj 


100.0% 




93.8% 








2 


i-H 

vD 

CO 


m 

vD 

CO 


vO 

CN 

IN 


i-H 

o 

\D 


o 

i-H 


o 

CN 


i-H 


vO 

CN 

IN 




CN 

SO 

I-H 


CO 

I-H 


00 

o 

I-H 


CN 

I-H 

I-H 


o 

I-H 


SO 

O 

I-H 


o 


so 

CN 

IN 




I-H 

00 

so 




1 * - In Dallas, one elementary school contains grades K - 3 and the other grades 4-6. 


** - Two elementary school sites are included in the calculations for these SoF sites. 
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TABLE 2 


Demographic Profile of Students Attending the SoF Middle Schools: 1990-91 
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* - Dallas middle school includes grades 7-8 only while middle schools in other cities include grades 6 through 8. 
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Appendix A 

SCHOOL OF T0I10HH0U CONCEPT PHPEH 

(as presented in 1990 to community leaders in the four cities) 



When the Hogg Foundation was established some 50 years ago, its charge was to develop "a 
broad mental health program for bringing great benefits to the people of Texas." Based on a 
recognition of schools’ access to children and an awareness of the potential of schools to help 
children and their families, the Foundation has undertaken a major new demonstration pro- 
ject, the School of Tomorrow. 

School of Tomorrow Concept 

For many youngsters in Texas the task of growing up is overwhelming. Children from single- 
parent families living in impoverished neighborhoods must deal on a daily basis with the prob- 
lems that many people know of about only through reading. Substance abuse, physical and 
sexual abuse, emotional abuse, dropping out of school, and teen pregnancy are just a few of 
the potential hurdles facing Texas youngsters. 

The School of Tomorrow is an innovative approach to helping solve many of the problems 
facing Texas children. It grew out of the work of Dr. James Comer of the Yale University 
School of Medicine and Dr. Edward Zigler of Yale's Bush Child Development Center, both of 
whom envision an expanded role for schools in meeting the needs of children. 

The concept of the School of Tomorrow is to provide an integrated array of health and human 
services, both treatment and prevention, using the school as the locus for their delivery. 
Essential features of the School of Tomorrow include: 

• The integration of a broad spectrum of health and human services in public schools. 

• Involvement of parents and teachers in the program activities. 

• Involvement of many organizations, both public and private, as partners. 

• A strong commitment to the project by superintendents, principals, and other school 
administrators. 

• A willingness to participate in the evaluation of the project. 

Hogg Foundation’s Role 

The Hogg Foundation for Mental Health is committed to demonstrating the effectiveness of 
the School of Tomorrow concept by funding demonstration programs in four sites — Austin, 
Dallas, Houston and San Antonio. In spring 1990 test sites received initial grants from the 
Foundation of $50,000 per year for five years — a total of one million dollars. The Foundation 
has set aside an equal amount to provide each site with technical assistance and evaluation 
support. For more information on the project, please call the Hogg Foundation. 
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BPPENOIX B, 

PHRENT CONSENT FORU 

(Active Consent Version) 



School of the Future Project 

Your child is invited to participate in a study that will focus on the mental and physical health 

of students in the Middle School. The Hogg Foundation for Mental Health has 

embarked on a project called the School of the Future in this school and others. We would 
greatly appreciate your child's participation in this survey. 

The Hogg Foundation has funded Elementary School and the related Middle 

School for at least a five year period to explore how schools can be improved, thereby enrich' 
ing the lives of children in this community. To accomplish this, local human service agen^ 
cies, public school systems, and teachers and parents are working together to improve your 
child's education and health. 

We hope to document the results of this project by collecting information on the students in 
these schools. The information your child shares with us will help school staff to understand 
the needs of their students better and should contribute to the development of programs 
offered by the project. It is also important to get a picture of the students and their needs, in 
order to assess any change that occurs over the history of the project. 

All 6th, 7th, and 8th graders enrolled in the School of the Future sites will be asked to com' 
plete surveys during the Fall of this school year. The interview should take approximately 50 
minutes (or one class period) to complete and explores issues such as self'CSteem, mental and 
physical welFbeing and their views on the school environment. Your child will be asked to 
participate in this survey at the school only once a year. 

After the surveys have been completed and collected, our evaluation staff will analyze the 
responses and develop a summary of the students at these schools. Results of the study will 
be made available to school personnel and parents in these school sites. 

A copy of the survey to be administered to your child is available for you to look over at the 
school library. If you have any questions about the survey or the School of the Future Project, 
please feel free to contact Project Coordinator's name at (xxx) xxx'xxxx or HF Research 
Associate's name at (xxx) xxx'Xxxx. 
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Dear Parent, 



It is important for you to know that any information obtained in connection with this study 
which can be identified with your child will remain confidential and will be disclosed only 
with your permission. Your decision whether or not to allow your child's participation in this 
survey will not prejudice your future relations with the University of Texas, the Hogg 

Foundation for Mental Health, or the Independent School District. Your child is 

under no obligation to participate in this study. If you do decide to allow your child to par- 
ticipate, he/she may withdraw at any time during the survey. 

You are making a decision whether or not to allow your child to participate in the School of 
the Future student self-report survey. Your signature indicates that you have read the infor- 
mation provided on the attached sheet and have decided to ALLOW your child's participa- 
tion. 

If you would like to have your child participate in the survey, please have your child return 
this form with your signature to his/her teacher. This will be taken as evidence of your will- 
ingness to allow your child's participation in the survey and your consent to have information 
used for purposes of the study. A copy of this consent form will be made available to you at 
any time in the future. 



Signature of Parent or Legal Guardian Date 



Signature of Investigator 



Date 



PflflENT CONSENT FOflN 

(Passive Consent Version) 



School of the Future Project 

Your child is invited to participate in a study that will focus on the mental and physical health 

of students in the Middle School. The Hogg Foundation for Mental Health has 

embarked on a project called the School of the Future in this school and others. We would 
greatly appreciate your child's participation in this survey. 

The Hogg Foundation has funded Elementary School and the related Middle 

School for at least a five year period to explore how schools can be improved, thereby enrich- 
ing the lives of children in this community. To accomplish this, local human service agen- 
cies, public school systems, and teachers and parents are working together to improve your 
child's education and health. 

We hope to document the results of this project by collecting information on the students in 
these schools. The information your child shares with us will help school staff to understand 
the needs of their students better and should contribute to the development of programs 
offered by the project. It is also important to get a picture of the students and their needs, in 
order to assess any change that occurs over the history of the project. 

All 6th, 7th, and 8th graders enrolled in the School of the Future sites will be asked to com- 
plete surveys during the Fall of this school year. The interview should take approximately 50 
minutes (or one class period) to complete and explores issues such as self-esteem, mental and 
physical well-being and their views on the school environment. Your child will be asked to 
participate in this survey at the school only once a year. 

After the surveys have been completed and collected, our evaluation staff will analyze the 
responses and develop a summary of the students at these schools. Results of the study will 
be made available to school personnel and parents in these school sites. 

A copy of the survey to be administered to your child is available for you to look over at the 
school library. If you have any questions about the survey or the School of the Future Project, 
please feel free to contact Project Coordinator's name at (xxx) xxx-xxxx or HF Research 
Associate’s name at (xxx) xxx-xxxx. 
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Dear Parent, 

It is important for you to know that any information obtained in connection with this study 
which can be identified with your child will remain confidential and will be disclosed only 
with your permission. Your decision whether or not to allow your child’s participation in this 
survey will not prejudice your future relations with the University of Texas, the Hogg 

Foundation for Mental Health, or the Independent School District. Your child is 

under no obligation to participate in this study. If you do decide to allow your child to par- 
ticipate, he/she may withdraw at any time during the survey. 

You are making a decision whether or not to allow your child to participate in the School of 
the Future student self- report survey. The return of this form with your signature indicates 
that you have read the information provided on the attached sheet and have decided to NOT 
ALLOW your child's participation. 

If you would like to have your child participate in the survey, you need not return this form. 
This will be taken as evidence of your willingness to allow your child's participation in the 
survey and your consent to have information used for purposes of the study. A copy of this 
signed consent form will be made available to you at any time in the future. 



Signature of Parent or Legal Guardian Date 



Signature of Investigator 



Date 
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Appendix C 

CHILD ASSENT FOHN 



School of the Future Project 



I agree to participate in a survey that is studying how children feel about their own physical 
and emotional welhbeing. As far as 1 know, my parents are aware of this study and they do 
not object to my taking part. 1 realize that 1 may refuse to participate and nothing bad will 
happen to me. 

When 1 sign my name to this page, 1 am indicating that this page was read to me and that I 
am agreeing to participate in this survey. My signature on this page shows that I understand 
what will be asked of me and that 1 may stop at any time. 



Signature of Student 


Date 


Printed Signature of Student 


School 


Signature of Investigator 


Date 
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TEHCHER CONSENT FORI1 



If You Wish To Participate, Return This Page 
BY March 11, 199„ 



It is important for you to know that any information obtained in connection with this study 
which can be identified with you or any of your students will remain confidential. Your deci' 
sion whether or not to participate in this survey will not prejudice your future relations with 
The University of Texas, the Hogg Foundation for Mental Health, or the 
Independent School District. 

You are under no obligation to participate in this study. If you do decide to participate, you 
may withdraw at any time. (However, you will only be compensated financially for the stu^ 
dents in your class that have been assessed using the surveys provided.) 

You are making a decision whether or not to participate in the Teacher Report Form surveys. 
Your signature indicates that you have read the information provided on the attached sheet 
and have decided to participate. 

If you would like to participate in the survey, please provide the information requested below, 
sign the form and return it to the school office. A copy of this consent form will be made 
available to you at any time in the future. 



Signature of Teacher 




Date 


Teacher Name (Printed) 




Social Security # 


Gender Ethnicity 


Highest Degree 
Attained 


Years Teaching 
Experience 


Name of School 


Classroom Grade 


# of Students in 
Class 


Home Street Address 


City, State, Zip Code 



^ — ature of Investigator ^ Date 
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